AD-A070  817 


UNCLASSIFIED 


ANALYTIC  SERVICES  INC  ARLINGTON  VA  F/6  5/2 

INTERVIEW  EVALUATION  OF  THE  DATASTAT  OUTPATIENT  PHARMACY  SYSTEM— ETC(U) 
JUL  78  J 0 H00600N  MOA903-78-C-0085 

ANSER-HSDN-78-2  NL 


riCATION  or  THIS  PAGE  (Whnt  Dmtm  Bnttrtd) 


REPORT  DOCUMENTATION  PAGE 


.^EPOHT 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


.^EPOHT  ni/MMA-2^ 

gSDN-78-2£ 


2.  GOVT  ACCESSION  NO. 

3. 

J 

1 

>.  PERFORMING  ORGANIZATION  NAME  AND  MORESS 

Analytic  Services  Inc.'' 

400  Army-Navy  Drive 
Arlington,  Virginia  22202 

It.  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

TRIMIS  Program  Office 
6917  Arlington  Road 
Bethesda,  Maryland  20014 


4.  MON/TORfNG  AGENCY  NAME  A ADORESS^if  diHefot  from  Conteotling 


0.  CONTRACT  OR  GRANT  NUMBERr*) 

I MD^j;y3-78-C-W5f 


10.  PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  6 WORK  UNIT  NUMBERS 


1$.  SECURITY  CLASS,  (of  thia  raport) 

UNCLASSIFIED 


tSfl.  OECLASSIPICATION/OOWNGRADING 

schedule 


16.  DISTRIBUTION  STATEMENT  (of  thfa  Raport) 


Approved  for  public  release;  distribution  unlimited 


17.  DISTRIBUTION  STATEMENT  (of  tha  abatrmet  entarad  tn  Block  20,  if  diffarant  from  Raport) 


It.  KEY  WORDS  fCondnu*  on  rayataa  aida  If  nacaaaary  and  idarxHty  by  block  numborj 

Evaluation:  Interview  Evaluation 

Information  Systems:  Automated  Information  System 

Medical  Pharmacies:  Military  Medical  Pharmacies 


20.  ABSTRACT  fContlnuo  on  fovoro#  afda  if  nacaaaary  and  fdantify  by  bloeJr  numborj 

This  report  presents  the  findings  of  interviews  conducted  in 
March  1978  with  the  pharmacy  staff  at  Naval  Regional  Medical 
Center  Charleston  to  obtain  their  subjective  assessments  of 
the  performance  and  Impacts  of  an  automated  pharmacy  information 
system  after  its  first  3 months  of  operation.  Seventeen  of 
the  20  pharmacy  staff  members  at  Charleston  were  interviewed, 
and  all  respondents  expressed  a favorable  opinion  of  the  system 


DD 


^ORM 

JAN  7»  WJ 


EDITION  OF  t NOV  li  OESOLETE 


UNCLASSIFIED ■ 

SECURITY  CLASSIFICATION  OF  THIS  PAGE  (mim  Dmi*  Bn<»r*d> 


HEALTH  SYSTEMS  DIVISION  NOTE 
HSDN  78-2 


INTERVIEW  EVALUATION  OF  THE 
DATASTAT  OUTPATIENT  PHARMACY  SYSTEM 
AT  NAVAL  REGIONAL  MEDICAL  CENTER  CHARLESTON 


Jonathan  D.  Hodgdon 


Approved  by 
Harry  E.  Emiet,  Jr. 

Vice  President-Health  Systems 


Conducted  for, 
and  in  cooperation  with, 
the  TRIMIS  Program  Office 
Assistant  Secretary  of  Defense  (Health  Affairs) 


Approved  tor  public  release 
distrihuhon  unfimiferf 


400  ArmyNavy  Drive.  ArMogton. Virginia  22202 


ABSTRACT 


'^he  DataStat  Outpatient  Pharmacy  System  is  designed  to 
improve  and  expand  pharmacy  operations  by  performing  such 
functions  as  the  printing  of  labels  and  reports,  patient 
profiling,  interaction  monitoring,  and  inventory  control. 
In  December  of  1977,  the  Naval  Regional  Medical  Center  at 
Charleston,  S.C.,  became  the  first  U.S.  military  hospital 
to  implement  DataStat.  This  paper  presents  the  results  of 
interviews  conducted  in  March  1978  with  the  pharmacy  staff 
at  NRMC  Charleston  to  obtain  their  subjective  assessments 
of  the  performance  and  impacts  to  date  of  the  automated 
outpatient  system.  ^ 

l\ 

The  following  list  summarizes  the  benefits  realized  to 
date  with  DataStat,  in  the  opinion  of  the  pharmacy  staff: 

• Benefits  to  Pharmacists 


Increased  involvement  in  profiling,  interaction 
monitoring,  and  compliance  monitoring 

Increased  involvement  as  therapy  advisers  to 
physicians 

- Improvements  in  amount  and  accessibility  of 
drug  information 

Increased  awareness  and  ability  to  detect 
potential  drug  problems 

Use  as  a management  tool 

Benefits  to  Technicians 

Increased  involvement  in  patient  care  process 

Increased  pharmacist  involvement  with  and 
supervision  of  technicians  ^ 

- Automatic  typing  of  labels  IV  ill 

- Use  as  an  educational  tool  | |®'--C]  f j I 
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• Benefits  to  Physicians 

Possible  changes  in  therapy  and  avoidance  of 
complications 

Improved  compliance  monitoring 
Resource  for  patient  information 
Resource  for  drug  information 

• Benefits  to  Patients 

Detection  of  potential  drug  problems 
Decreased  time  required  to  refill  prescriptions 
Improved  label  accuracy  and  legibility 

• Benefits  to  Others 

Patient  information  resource  for  hospital 
Drug  information  resource  for  command. 

Benefits  not  realized  according  to  the  respondents  are  « 

1)  reductions  in  patient  waiting  time  (may  have  increased) , 

2)  reductions  in  new  technician  training  time  (may  have  in- 
creased), 3)  reductions  in  time  required  to  print  labels, 
and  4)  the  use  of  and  associated  benefits  of  DataStat  in 
the  area  of  inventory  control.  In  addition,  numerous  capa- 
bilities of  DataStat  were  reportedly  not  being  utilized  to 
their  full  potential  as  a result  of  1)  perceived  understaffing, 
2)  physician  unfamiliarity  with  DataStat,  and  3)  a large  pro- 
portion of  patients  not  yet  logged  into  the  system. 

Overall  the  system  has  been  favorably  received  by  the 
entire  pharmacy  staff,  and  all  those  interviewed  perceived 
it  as  having  a positive  impact  on  the  quality  of  patient 
care.  They  have  encountered  no  major  problems  with  the  hard- 
ware equipment  or  the  service  provided  by  the  lessor.  However, 


slow  terminal  response  times,  software  problems,  and  staff 
resistance  to  change  were  encountered  during  the  first  month 
of  operation.  The  majority  of  patient  comments  about  the 
system  seem  to  be  negative  but  become  less  negative  after 
a staff  member  explains  the  functions  performed  by  the  system 
and  the  temporary  circumstances  contributing  to  patient  waiting 
time. 

Suggestions  for  improvements  offered  by  the  respondents 
include  1)  using  a faster  label  printer,  2)  maintaining  the 
same  or  greater  staffing  level  with  DataStat,  3)  logging 
patients  into  the  system  elsewhere  or  earlier,  4)  not  in- 
stalling the  system  during  the  busy  winter  months,  and  5) 
providing  better  training  and  user  instructions. 
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I.  ACCEPTANCL  OF  DATASTAT  AND  IMPROVEMENTS  SUGGESTED 
BY  THE  PHARMACY  STAFF 

A.  Introduction 

In  mid-September  1977  the  first  cathode  ray  tube  (CRT) 
terminal  was  installed  in  the  pharmacy  at  the  Naval  Regional 
Medical  Center  (NPJMC)  , Charleston,  S.C.,  in  order  to  begin 
logging  patients  into  the  computer  files  of  the  DataStat  Out- 
patient Pharmacy  System.  The  system  became  fully  operational 
at  Charleston  on  December  1,  1977.  About  3 months  later,  on 
March  7 and  8,  1978,  interviews  were  conducted  with  the  phar- 
macy staff  to  obtain  their  perceptions  of  the  impacts  and 
performance  of  the  DataStat  System.  This  paper  reports  the 
principal  findings  and  insights  gained  from  these  interviews. 

As  of  the  second  wee)c  in  March  the  pharmacy  staff  con- 
sisted of  four  pharmacists  and  16  technicians  (techs) . Three 
technicians  were  not  interviewed  (one  was  at  another  dis- 
pensary, one  was  on  leave,  and  one  was  the  overnight  tech), 
leaving  a total  of  17  respondents.  The  interviews  were  con- 
ducted in  the  pharmacists'  offices,  with  only  the  interviewer 
and  the  respondent  present  at  each  session.  While  appointments 
were  not  scheduled  in  advance,  no  interview  was  either  rushed 
or  abbreviated  for  any  reason.  Because  of  the  unstructured 
nature  of  the  interview,  tape  recorders  were  used  with  the 
permission  of  the  - respondent.  The  respondents  were  assured 
of  the  impartiality  of  the  interviewer  and  of  their  own  ano- 
nymity. In  the  analysis  that  follows,  respondents  are  identi- 
fied by  numbers  randomly  assigned  after  all  interviews  were 
completed. 

The  technician  interviews  ranged  in  duration  from  15 
minutes  to  40  minutes  and  averaged  about  30  minutes.  Two 
pharmacist  interviews  lasted  1 1/2  hours,  another  took  1 1/4 

1 


hours,  and  thv®  fourth  took  1-3/4  hours.  The  questions  asked 
in  the  interviews  were  those  presented  in  the  draft.  Evaluation 
Plan  for  an  Outpatient  Pharmacy  System  (DataStat) , Analytic 
Services  Inc.,  July  1977.  The  two  interview  schedules  that 
were  used  are  presented  in  Section  II.  Neutral  probes  were 
supplemented  by  focused  probes  designed  to  steer  the  respondent 
back  to  the  main  topic  or  response  categories  without  biasing 
his  responses.  While  responses  were  unusually  uniform  and 
positive  after  only  3 months  of  automated  data  processing 
(ADP)  operations,  the  staff  were  not  informed  beforehand  of 
the  Interview  Evaluation  Instrument  (lEI)  contents,  and  it  is  the 
opinion  of  the  interviewers  that  the  respondents  were  giving 
their  own  personal  and  candid  assessments  of  DataStat.  Finally, 
it  should  be  pointed  out  that  from  the  time  of  DataStat  im- 
plementation to  the  time  interviews  were  conducted,  the  phar- 
macy staffing  level  was  reduced  slightly  (not  a consequence 
of  DataStat)  and,  with  the  flu  season,  the  patient  load  in- 
creased from  about  750  to  over  1,000  prescriptions  per  day. 

These  two  changes  had  a bearing  upon  the  performance  of  the 
pharmacy  and  the  computer,  as  will  become  clear  in  the  pages 
that  follow. 

B.  Pharmacist  Interviews 
1 . Drug  Interactions 

The  first  series  of  questions  on  the  pharmacist  lEI  ad- 
dressed the  impact  of  DataStat  in  the  area  of  drug  interactions. 
All  four  respondents  reported  that  their  involvement  in  drug 
interaction  monitoring,  detection  (including  allergies) , and 
counseling  with  physicians  had  increased  from  near  zero  to 
daily  involvement  in  each  activity,  and  that  this  was  a direct 
result  of  DataStat  implementation.  The  magnitude  and  impor- 
tance of  these  new  tasks  to  the  pharmacist  can  be  better  seen 
by  some  of  their  comments  below: 
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Before  [DataStat]  we  had  no  way  of  monitoring 
drug  interactions.  Well,  there  are  a couple 
of  classic  drug  interactions  that  everybody 
knows  in  pharmacy,  but  the  only  way  you  got 
to  catch  those  was  if  the  two  prescriptions 
were  presented  at  the  same  time.  [Otherwise] , 
we  had  no  chance  of  ever  comparing  prescrip- 
tion A with  prescription  B.  (Pharmacist  #4) 

...and  drug  allergies  of  course,  we  didn't  even 
know  before  because  we  didn't  get  the  patient's 
chart  and  the  chart  is  what  has  "allergic  to 
penicillin"  in  it.  We  never  saw  that.  But 
now  we  see  it  because  we  inputted  that  data 
into  the  computer  and  immediately  the  com- 
puter tells  us  "allergic  to  penicillin"  as  it 
comes  up.  Before,  we  just  had  to  give  what 
the  doctor  wrote,  with  the  hopes  that  he  caught 
the  allergies.  (Pharmacist  #2) 

This  has  precipitated  a lot  more  calls  to  the 
physician  than  we  aver  did  before.  But  that's 
good  for  us  to  communicate  with  them.  And  once 
we  tell  them  about  this  interaction  between  the 
two  drugs  the  patient's  taken,  then  they'll  know 
it  for  the  future...!  think  it  sort  of  educates 
the  physician  because  half  the  time  when  we  call, 
the  doctor  says,  "I  didn't  know  those  interacted," 
or  "I  didn't  know  she  was  taking  that'.'  (Pharmacist  #2) 

Our  role  has  changed  in  that  we  have  become  more 

of  a therapy  advisor  than  in  the  past.  (Pharmacist  #3) 

Tables  1 and  2 summarize  the  benefits  and  disadvantages  of 
DataStat  in  the  area  of  drug  interactions  as  mentioned  by 
the  pharmacists.  The  advantages  most  often  reported  were 
that  DataStat  made  interaction  monitoring  possible  in  the 
first  place  by  profiling  and  acting  as  a fifth  pharmacist, 
that  the  pharmacy  staff  (especially  technicians)  were  now 
more  aware  of  drug  interactions,  that  potential  lawsuits 
can  be  avoided  (benefitting  the  physician  and  Patient  Affairs) , 
and  that  DataStat  serves  as  a handy  source  of  drug  interaction 
information. 


TABLE  1 

PHARMACIST  RESPONSE  TO  QUESTION 
ON  EFFECT  OF  DATASTAT  ON  DRUG  INTERACTION 
MONITORING  SYSTEM-BENEFITS 


Response  Category 


Monitors  for  pharmacists 
where  not  feasible  before 

Increases  staff  awareness 
of  drug  interactions 

Avoids  potential  lawsuits 
Serves  as  information  source 

Permits  faster  response  to 
inquiry 

Compensates  for  understaffing 

Involves  pharmacists  more  at 
front  line  (filling  line) 


Number  of  Responses 


TABLE  2 

PHARMACIST  RESPONSE  TO  QUESTION 
ON  EFFECT  OF  DATASTAT  ON  DRUG  INTERACTION 
MONITORING  SYSTEM-DISADVANTAGES 


All  pharmacists  mentioned  one  shortcoming  of  the  DataStat 
drug  interaction  system;  namely,  the  failure  of  the  system  to 
distinguish  between  medications  prepared  for  internal  and  ex- 
ternal use;  which  leads  to  numerous  unnecessary  interaction 
reports  and  to  Pharmacist  #4's  comment: 

...what  I'm  afraid  of  is  that  if  I continue  to 
ignore  some  things  that  I know  are  superfluous, 

1 will  get  into  the  ignoring  habit  and  I don't 
want  to  get  into  an  ignoring  cycle,  if  there  is 
such  a term. 

Two  respondents  offered  the  opinion  that  there  was  some  unused 
system  potential  in  this  area  due  to  the  lack  of  manpower 
available  to  follow  up  each  potential  interaction.  This  is 
the  first  of  numerous  instances  where  the  staff  (including 
technicians)  reported  underutilization  of  DataStat  capacity 
as  a direct  result  of  perceived  understaffing. 

Respondents  were  asked  (later  in  the  interview)  if  they 
discussed  drug  interactions  with  prescribers  and  how  they 
thought  the  prescribers  viewed  DataStat 's  capabilities  in 
this  area.  All  four  pharmacists  reported  at  least  weekly 
drug  interaction  discussions  with  physicians  and  seemed  to 
agree  that  while  many  physicians  utilized  the  pharmacy  more 
and  changed  therapy  more  since  DataStat,  many  others  were 
still  unaware  of  the  system  and/or  its  capabilities. 

1 think  that  they  are  in  the  process  of  perceiving 
more,  of  noticing  that  we  are  contacting  them  more 
frequently  about  drug  interactions.  (Pharmacist  #3) 

If  1 had  to  say  which  we  get  more  of,  the  change  in 
therapy  or  a "thanks  for  calling  but  let  them  go 
anyway,"  1 think  we  get  more  changes  in  therapy. 

Another  thing,  of  course,  that  happens  to  us  is 
this,  "I  told  him  to  stop  taking  that  other  medi- 
cation" or  "she's  not  taking  that  anymore,"  that 
sort  of  thing. . . But  there  are  more  positive 
responses  to  our  calls  than  negative.  (Pharmacist  #4) 
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We  have  already  caught  12  to  15  potential  allergy 
problems  since  DataStat,  and  called  the  physician. 

And  that's  only  allergies.  (Pharmacist  #1) 

In  summary,  the  pharmacists  felt  that  the  benefits  of 
DataStat  in  the  area  of  drug  interaction  monitoring  and 
detection  far  outweighed  the  disadvantages  of  the  system  in 
this  area  and  that  current  problems  were  correctable  soft- 
ware problems.  Primarily,  the  existence  of  the  DataStat 
drug  interaction  system  has,  in  the  opinion  of  the  pharma- 
cists, enabled  them  to  become  more  involved  with  both  the 
physicians  and  the  pharmacy  techs,  called  upon  their  pro- 
fessional knowledge  more  often,  and  subsequently  enhanced 
their  ability  to  provide  quality  health  care. 

2 . Patient  Profiles 

Another  new  role  for  the  pharmacists  at  NRMC  Charleston 
was  working  with  the  patient  profile,  as  all  respondents 
agreed  that  no  profiling  was  done  outside  of  the  patient's 
chart  under  manual  operations,  nor  was  it  feasible.  The 
four  officers  also  agreed  that  the  existence  of  the  drug 
profile  has  increased  the  extent  of  their  involvement  in 
the  health  care  delivery  process  and  definitely  improved 
the  quality  of  health  care.  It  is  from  the  profile  that 
most  other  DataStat  benefits  are  derived;  namely,  detecting 
interactions,  overlaps  (patient  receives  same  drug  from 
two  physicians) , allergies,  medication  stockpiling,  etc. 
Another  benefit  of  the  profile  to  physicians  is  pointed  out 
by  Pharmacist  #2; 

The  military  changes  doctors  every  2 or  3 
years.  No  matter  what  doctors  were  there 
with  them  before,  [with  DataStat]  the  new 
doctor  can  be  informed  of  what  the  patient 
has  been  on  from  the  past,  and  how  the 
patient  is  going  with  regard  to  medication. 
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Three  of  the  four  respondents  felt  that  their  professional 
self-image  had  been  improved  as  a result  of  their  profile- 
related  tasks.  The  fourth  did  not  perceive  any  change  in  self- 
image  but  agreed  with  the  others  that  each  of  them  receives 
physician  inquiries  almost  daily  for  profile-related  informa- 
tion. More  often  than  not,  the  physicians  examine  the  profiles 
themselves  according  to  the  pharmacists. 

3.  Counseling  and  Compliance 

While  interaction  monitoring  and  detection  were  perceived 
as  an  important  DataStat  benefit  by  the  pharmacists,  they  re- 
ported that  the  more  common  day-to-day  assistance  from  the 
computer  was  provided  in  the  area  of  patient  compliance.  The 
four  officers  felt  that  DataStat  had  not  only  facilitated 
what  compliance  monitoring  existed  before  (by  printing  a 
projected  refill  date  on  the  label)  but  here  again  enabled 
them  to  add  new  tasks;  namely,  checking  for  many  more  types 
of  drug  misuses  than  in  the  past.  Overlap  prescriptions 
from  different  physicians  (duplications)  were  now  being 
detected  fairly  often,  and  instances  of  stockpiling  detection 
were  not  uncommon,  especially  since  military  patients  usually 
receive  free  medications: 

...that  would  never  happen  in  a civilian  practice, 
never.  Patients  just  would  not  pay  for  [extra  med- 
ication]. Here,  with  no  one  paying  for  it,  with 
the  fact  that  it's  hard  to  get  an  appointment  with 
a physician,  and  with  the  fact  that  they  do  have  to 
wait  in  line,  they  try  to  get  as  much  as  they  can. 

What  we  can  do  here  is  we  can  monitor  to  see  how 
much  a patient  is  t2dclng,  and  to  see  how  much  they 
have  at  home.  These  patients  stock  up  so  much  med- 
ication it  borders  on  the  point  of  being  ridiculous. 

(Pharmacist  #1) 

All  four  respondents  perceived  improvements  in  their 
ability  to  detect  overdue  or  early  refills  with  DataStat; 


however,  three  also  mentioned  that  they  were  not  utilizing 
its  ability  to  detect  overdue  refills  (i.e.,  obtaining  a 
list  from  the  computer  of  those  who  missed  their  scheduled 
refills  that  day  or  week,  and  consulting  the  physician)  for 
two  reasons: 

We  don't  have  enough  staff  to  perform  this  function, 
number  one;  and  number  two,  we  feel  that  we  haven't 
yet  purged  the  stockpile  of  medications  that  the 
patients  have  saved  up  from  before.  (Pharmacist  #4) 

Understaffing  was  also  cited  by  two  officers  as  the  reason 
that  numerous  overlaps  and  instances  of  apparent  stockpiling 
were  still  being  missed  or  detected  too  late  (i.e.,  they  were 
not  able  to  call  up  and  check  every  profile  before  filling  a 
prescription*) . All  four  officers  called  understaffing  the 
main  reason  that  patient  counseling  by  pharmacists  has  con- 
tinued at  a low  level  of  involvement. 

The  pharmacists  agreed  that  patient  counseling  was  part 
of  their  work  in  the  pharmacy;  however,  only  two  felt  that 
DataStat  had  increased  their  involvement  in  this  activity 
(the  other  two  reported  no  change) . Although  all  felt  that 
the  system  had  the  capacity  to  improve  patient  counseling, 
two  respondents  mentioned  that  in  addition  to  the  staffing 
problem,  there  was  no  counseling  window  or  booth  where  the 
pharmacist  and  patient  could  talk  in  private. 

The  pharmacists  had  all  discussed  patient  compliance 
with  prescribers  and  were  evenly  divided  when  asked  whether 
or  not  they  thought  physicians  were  noticing  changes  in  com- 
pliance since  DataStat.  Three  of  the  four,  however,  felt 
that  physicians  were  already  acknowledging  the  capacity  of 
DataStat  to  change  patient  compliance. 

* Almost  all  prescriptions  are  filled  using  what  is  called 
the  advanced  function  of  DataStat.  In  order  to  determine 
whether  or  not  a reported  overlap  is  a true  misuse  of 
medication,  the  staff  member  must  escape  from  (leave) 
this  function  and  call  up  the  Individual's  profile. 
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4 . Drug  Information 


Responses  to  the  first  lEI  drug  information  question  were 
related  to  discussions  with  prescribers  about  drug  interactions 
and  have  already  been  presented  under  that  topic  heading.  Re- 
sponses to  the  second  question  again  indicate  a consensus  that 
DataStat  has  changed  the  drug  information  system  at  NRMC  Charleston 
by  making  more  information  available  more  readily,  especially 
for  the  technicians.  Typical  of  the  comments  received  was  that 
of  Pharmacist  #3  who  commented  that  in  his  view  the  primary 
impact  of  DataStat  on  drug  information  was  that  information 
now  appeared  "in  one  big  book  versus  many  smaller  books  before.” 

All  four  officers  felt  that  the  pharmacy  had  already  been 
and  will  continue  to  be  improved  by  the  DataStat  drug  infor- 
mation system  for  reasons  similar  to  those  listed  in  Table  1. 

They  also  felt  that  there  were  numerous  benefits  of  DataStat 's 
drug  information  outside  the  pharmacy,  and  these  responses 
are  presented  in  Table  3. 

Pharmacist  #2  describes  the  second  benefit  listed  in 
more  detail: 

The  billing  office  comes  to  us  for  the  patient's 
correct  mailing  address;  the  GP  clinic  called  ten 
times  just  last  Saturday  for  the  new  addresses 
and  phone  nianbers  of  recent  transfers  into  this 
area  whom  they  have  treated;  eind  doctors  come  in 
here  all  the  time  for  the  correct  home  phone 
numbers  for  some  of  their  patients. 

It  is  also  interesting  to  note  the  benefit  reported  by  one 
pharmacist — that  with  DataStat  they  will  know  the  catchment 
area  or  number  of  patients  eligible  for  care  at  their  hos- 
pital as  soon  as  the  number  of  outpatients  appearing  at  the 
window  without  a red  "C"  marked  on  their  ID  card  (C  indicates 
logged  in  DataStat)  decreases  to  near  zero. 
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TABLE  3 

PHARMACIST  RESPONSE  TO  QUESTION 
ON  BENEFITS  OF  DATASTAT  DRUG  INFORMATION 
OUTSIDE  THE  PHARMACY 


Response  Category 

Number  of  Responses 

Improves  patient  care  through 
changes  in  therapy,  etc. 

4 

Serves  as  most  current  hospital 
source  for  patient  addresses 
and  phone  numbers 

3 

Assists  the  physician  as  well  as 
the  pharmacy 

2 

Serves  physicians  and  Patient 
Affairs  in  avoiding  potential 
lawsuits 

2 

Serves  as  more  of  a drug 
information  center  for  the 
command  than  before 

1 

Will  soon  tell  us  the  number 
of  patients  eligible  for  care 
at  this  hospital 

1 

Will  pay  for  itself  econom- 
ically in  the  long  run 

1 

The  pharmacists  reported  that  the  pharmacy  was  only  begin- 
I ning  to  experiment  with  DataStat's  inventory  function  in  the 

area  of  controlled  substances  and  that  DataStat  had  not  yet 
j had  any  impact  in  the  inventory  area.  The  main  obstacle  to 

, be  overcome  is  that  at  NRMC  Charleston  the  units  of  measure 
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for  inventory  are  different  from  the  units  of  purchasing 
and  the  system  cannot  yet  accommodate  this.  All  of  the 
respondents,  however,  were  optimistic  that  DataStat  would 
be  able  to  perform  some  valuable  inventory  functions  in 
the  future  and  looked  forward  to  benefiting  from  this  large 
source  of  unused  system  potential. 

6.  Outpatient  Dispensing  Methods 

The  pharmacists  unanimously  agreed  that  DataStat  had 
changed  the  patient  encounter  at  the  window  and  that  patient 
waiting  time  for  routine  prescriptions  had  not  decreased. 
Regardless  of  the  direction  of  computer  impact,  the  seasonal 
increase  in  patient  demand  and  the  reduction  in  staffing 
served  to  increase  patient  waiting  time.  While  most  of  the 
patients  were  quick  to  "put  two  and  two  together  and  blame 
the  computer"  (Pharmacist  #3),  the  four  officers  could  not 
agree  as  to  whether  or  not  DataStat  had  a net  impact  on 
waiting  time. 

Computer-related  tasks,  such  as  checking  to  see  if  the 
patient  is  in  the  computer,  checking  the  profile  for  over- 
laps or  any  evidence  of  drug  misuse,  and  calling  the  physi- 
cian in  cases  of  potential  allergy  or  interaction  problems 
all  tend  to  add  to  the  patient's  wait.  An  even  more  important 
new  source  of  delay  was  the  fact  that  from  December  1977  to 
March  1978  the  majority  of  outpatients  were  not  logged  into 
the  computer  and  still  had  to  fill  out  the  computer  form. 

On  the  other  hand,  the  new  labels  made  scm\e  aspects  of  out- 
patient dispensing  easier  and  faster;  in  particular,  the 
time  required  to  refill  prescriptions.  Two  respondents 
felt  that  DataStat  had  Increased  the  patient  waiting  time, 
while  the  other  two  perceived  no  net  change  and  attributed 
patient  cOTiplaints  to  their  expectation  of  shorter  waits 


coupled  with  an  increase  in  their  demands  for  services  and 
a decrease  in  staffing.  The  following  comment  echoes  the 
concern  of  several  staff  members  with  patient  waiting  time; 

Something  needs  to  be  done  to  shorten  the  patient 
waiting  time— that's  my  major  complaint  about  the 
system.  I feel  guilty  when  I see  little  kids  out 
there  having  to  wait  with  their  parents — sick, 
feverish,  having  to  wait  for  their  prescriptions. 

I feel  very  bad  about  this,  and  the  techs  feel 
the  same  way.  (Pharmacist  #3) 

The  second,  third,  and  fourth  interview  questions  in  the 
lEI  outpatient  dispensing  section  asked  the  respondent  to 
compare  automatic  label  preparation  with  manual  label  prep- 
aration. The  only  point  of  disagreement  concerned  the  time 
required  to  prepare  labels.  Two  pharmacists  felt  DataStat 
had  increased  the  time  required,  while  the  other  two  reported 
a decrease  in  label  preparation  time.  Part  of  the  difficulty 
in  drawing  a conclusion  here  appears  to  be  due  to  changes  in 
the  way  labels  are  prepared.  One  person  enters  label  infor- 
mation now  compared  with  two  or  three  typists  before;  DataStat 
labels  are  printed  in  batches  of  15  to  25  and  in  triplicate 
now  compared  with  individual  typing  before;  and  DataStat 
now  runs  most  of  its  checks  for  interactions,  overlaps,  etc., 
prior  to  printing  the  labels.  The  pharmacists  did  agree 
that  during  peak  workloads,  the  300-baud  (30  characters  per 
second)  label  printer  appeared  to  be  a bottleneck  that  could 
be  eliminated  by  a faster  printer. 

All  respondents  reported  dramatic  improvements  in  both 
label  accuracy  and  legibility,  with  the  virtual  elimination 
of  spelling  errors,  incomplete  information,  cross-outs,  type- 
overs,  and  lost  prescriptions.  The  respondents  mentioned 
fewer  lost  prescriptions  as  a result  of  having  the  label 


printed  in  three  parts,  all  with  the  correct  prescription 
nvimber;  part  1 for  the  bottle  itself,  part  2 for  the  pre- 
scription blank,  and  part  3 as  a bag  label  for  those  patients 
returning  later  to  pick  up  medications. 


Finally,  in  regard  to  labels,  the  respondents  were  asked 
to  compare  the  amount  of  information  on  the  new  label  with 
that  on  the  old  labels  that  were  prepared  manually.  All  re- 
ported an  increase  in  the  amount  of  information  and  cited 


the  projected  refill  date  and  the  three-part  label  printing 
as  being  improvements  associated  with  DataStat.  They  per- 
ceived the  prescription  check  process  (ensuring  that  the 
prescription,  the  label,  and  the  medication  correspond)  as 
being  essentially  unchanged,  except  that  improvements  in  label 
accuracy  and  computer  numbering  of  the  three  label  parts  had 
reduced  the  frequency  of  errors  and  lost  prescriptions. 


When  asked  if  DataStat  had  changed  any  other  drug  dis- 
pensing activities,  two  pharmacists  mentioned  additional 
differences  in  label  preparation.  For  any  given  label  a tech- 
nician need  only  type  a few  codes  and  abbreviations,  and  the 
computer  will  do  the  rest.  Because  labels  were  being  printed 
in  batch,  however,  this  meant  that  they  had  to  be  paired  with 
the  original  written  prescriptions  which  could  result  in 
mismatching — ^unlike  the  manual  method  of  typing  labels  from 
the  prescription. 

In  svimmary,  the  pharmacists  were  pleased  that  they  were 
"definitely  preparing  a more  professional  product"  (Pharmacist 
#4)  that  was  being  delivered  to  the  patient  more  carefully, 
legibly,  amd  accurately;  however,  they  were  disappointed  in 
DataStat' s ability  to  reduce  patient  waiting  time  at  the 
window.  As  the  flu  season  passes  and  the  majority  of  patients 
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become  logged  into  the  computer,  the  problem  of  patient  waiting 
time,  according  to  the  pharmacists,  should  be  less  severe. 

■ 7 . General  Questions 

j The  four  officers  were  asked  if  DataStat  had  impacted 

j any  other  professional  activities.  One  respondent  mentioned 

the  increased  involvement  of  the  pharmacists  with  technicians 
on  the  front  line.  Two  others  described  some  problems  with 
personality  clashes  and  resistance  to  change  in  the  pharmacy 
during  DataStat 's  first  month  of  operation.  Reasons  given 
for  the  initial  negative  impact  of  the  computer  on  staff  re- 
I lations  were;  1)  everyone  was  required  to  work  extra  hours 

j to  make  the  system  succeed;  2)  the  technicians  were  not  al- 

I lowed  to  use  the  CRTs  while  the  system  was  so  vulnerable  to 

' user  error;  emd  3)  as  patient  waiting  time  steadily  increased, 

staff  members  began  to  question  the  merits  of  DataStat. 

Three  of  the  four  pharmacists  reported  an  increase  in 
new  technician  training  time  because  of  DataStat,  as  explained 
by  Pharmacist  #4: 

? 

I It  tcikes  more  time  to  teach  a person  how  to  use 

I the  computer  than  it  does  to  teach  him  how  to  type 

because  they  know  how  to  type  when  they  come  into 
I the  pharmacy.  And  so,  yes,  certainly  there's  been 

an  increase  in  training  time. 

i The  fourth  felt  that  the  observation  period  required  to  mon- 

* itor  the  performance  of  a new  technician  was  still  the  same 

length  of  time  as  before  DataStat.  Additional  comments  were 
that  only  a couple  of  technicians  were  new  since  DataStat  and 

J that  the  training  process  now  seemed  more  like  a training/ 

education  process  with  DataStat. 

The  respondents  were  generally  very  pleased  with  the  per- 

• formance  and  capabilities  of  the  hardware  equipment  and  with 
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the  service  provided  by  the  lessor.  Terminal  response  times 
were  reported  to  be  usually  very  good;  however,  one  pharmacist 
felt  that  during  peak  hours  the  DataStat  capabilities  at  the 
central  hardware  site  were  inadequate.  Another  respondent 
commented  that  an  additional  CRT  would  be  highly  beneficial, 
but  only  if  accompanied  by  an  increase  in  staff . 

No  one  indicated  problems  in  the  area  of  computer  printing 
of  reports.  The  officers  mentioned  that  reports  could  be 
generated  without  delay  whenever  desired  and  that  the  formats 
and  amount  of  information  presented  were  also  appropriate  for 
the  needs  of  the  pharmacy.  Some  reports  were  not  yet  being 
utilized;  namely,  those  relating  to  inventory  and  to  patient 
compliance  evaluation  {no  shows  for  refills,  etc.),  but 
these  would  be  put  to  use  in  the  near  future,  according  to 
the  respondents.  Reports  used  daily  in  the  pharmacy  are  the 
lists  of  command  (sig)  codes  and  drug  abbreviations  and  the 
end-of-day  summary  containing  a list  of  what  prescriptions 
were  filled  that  day,  for  whom,  by  whom,  at  exactly  what  time, 
etc. 

8 . Pharmacists*  Overall  Opinions  of  DataStat  and  Suggestions 
for  Improvement 

In  general,  the  DataStat  Outpatient  Pharmacy  System  has 
been  received  very  favorably  by  all  four  pharmacists  at  NRMC 
Charleston.  Their  svmunary  statements  speak  for  themselves: 

I would  say  I'm  very  satisfied  with  it.  I think 
it's  really  an  exciting  concept  and  I think  it's 
really  good.  I think  the  potential  for  utilizing 
one  of  these  systems  is  there,  and  that  we're  not 
utilizing  it  enough.  1 don't  know  if,  with  manpower 
being  what  it  is  in  the  military,  it  ever  could  be 
[fully  utilized].  (Pharmacist  #1) 
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I'm  sold  on  it.  I was  sold  on  it  when  I first  saw 
it  7 years  ago...  I didn't  realize  all  the  problems 
that  might  come  up  with  it,  but  I'm  glad  we  got  it. 
I'm  glad  I was  a part  of  it  and  I hope  that  we  go 
Navy-wide  with  it... but  it's  going  to  take  a lot  of 
management  effort.  (Pharmacist  #2) 

I'm  100  percent  favorable  for  the  concept  and  im- 
plementation of  it.  I'm  very  much  for  the  system 
and  hope  it  grows,  flourishes,  and  becomes  universal. 
I also  hope  that  in  future  facilities  that  get  this 
system,  someone  will  look  at  it  and  say  "Well  now, 
we're  going  to  be  turning  out  more  work  here,  gener- 
ating more  information,  so  we're  going  to  need  more 
people."  Here  we  didn't  get  any  more  people,  and 
that's  been  a problem.  (Pharmacist  #3) 


I'm  very  happy  with  the  system.  I think  that  we've 
demonstrated  that  a computerized  outpatient  pharmacy 
system  can  work  in  a 1 , 000-prescription-a-day  oper- 
ation and  do  more  than  just  print  labels,  because 
obviously  there  are  a lot  of  ways  to  print  labels 
automatically  faster  and  better,  i.e. , just  a mag 
card  typewriter.  But  I think  we're  doing  a lot  more 
than  just  printing  labels  and  that's  the  biggest 
thing  I hope  we  Cein  impress  upon  our  customers  and 
in  fact  upon  the  people  within  the  pharmacy,  too, 
because  I think  there  are  a lot  of  people  within  the 
pharmacy  who  look  at  it  as  just  a glorified  type- 
writer. It  is  more  than  that  and  I think  that  our 
technic iems  now  are  becoming  aware  of  that. 
(Pharmacist  #4) 


At  the  close  of  the  interview,  respondents  were  asked 
for  any  suggestions  to  improve  DataStat  or  its  implementation. 
Table  4 lists  the  responses  given  by  the  four  pharmacists. 

All  of  the  pharmacists  suggested  a longer  period  for  logging 
patients  into  DataStat  than  the  2 1/2  months  at  NRMC 
Cheurleston,  When  the  system  was  brought  up  and  running, 
less  than  one-third  of  the  patient  population  was  entered  in 
the  computer,  according  to  one  pharmacist.  Three  respondents 
also  recommended  logging  patients  into  the  system  from  a 
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TABLE  4 

PHARMACIST  SUGGESTED  IMPROVEMENTS  TO 
DATASTAT  OR  ITS  IMPLEMENTATION 


Response  Cetegory 


Number  of  Responses 


Begin  logging  patients  into 
system  earlier 


Increase  staffing  level  with 
new  system 


Have  drug  interaction  software 
discriminate  between  medica- 
tion prepared  for  internal  and 
external  use 


Use  faster  label  printer 


Log  patients  into  system 
elsewhere  in  hospital 


Install  in  off  season  rather  than 
during  busy  winter  months 


Create  an  in-service  imple- 
mentation team  to  assist 
individual  sites 


Involve  technicians  with 
system  more  actively 


Formalize  user  training/ 
improve  vendor  instructions 


Improve  time-sharing  capabilities 
at  central  hardware  site 


Reference  drug  interaction 
reports 


Generate  more  statistical 
reports 


Use  less  tiring  color  back- 
ground for  CRT  display 


Use  sig  base  and  drug 
abbreviations  formula 
developed  at  Charleston 
for  future  sites 


terminal  elsewhere  in  the  hospital.  The  officers'  suggestion 
to  increase  the  staffing  level  with  OataStat  relates  to 
their  desire  to  use  more  of  the  system's  potential  and  to 
Pharmacist  #3's  comment  that  "the  pharmacist  shouldn't  have 
to  come  in  at  5:00  a.m.  just  to  log  in  patients." 

The  respondents  repeated  their  suggestions  that  the  Data- 
Stat  drug  interaction  software  be  modified  to  distinguish 
between  medications  prepared  for  internal  and  external  use, 
and  that  a faster  printer  be  used  to  handle  the  peak  loads 
of  over  1,000  prescriptions  per  day  encountered  at  NRMC 
Charleston.  Suggestions  to  install  the  system  during  the 
off  season  rather  than  the  busy  season,  to  send  an  implementa- 
tion team  ("preferably  in-service  with  experienced  operators 
and  a public  relations  person")  to  new  sites,  and  to  get 
the  technicians  more  involved  with  the  system  also  were 
mentioned  by  more  than  one  pharmacist. 

C.  Technician  Interviews 

The  13  technicians  interviewed  were  asked  about  their 
personal  reactions  to  DataStat,  its  ease  of  use,  its  impact 
on  their  jobs,  and  their  suggestions  for  improvements. 

1.  Reaction  to  System 

All  technicians  interviewed  expressed  favorable  overall 
opinions  of  DataStat.  Eleven  of  the  13  technicians  reported 
that  they  liked  using  the  system;  the  other  two  considered 
it  no  more  or  less  enjoyable  to  work  with  than  the  manual 
system.  Table  5 summarized  the  responses  of  the  technicians 
when  asked,  "What  do  you  like  about  the  system?" 
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TABLE  5 

TECHNICIAN  RESPONSE  TO  QUESTION 
"WHAT  DO  YOU  LIKE  ABOUT  THE  SYSTEM?" 


Response  Category 

Number  of  Responses 

Results  in  fewer  spelling 
errors/retypes 

6 

Includes  built-in  safeguards 

6 

Codes  and  abbreviations  make 
for  ease  of  use 

6 

Does  most  of  necessary  typing 

5 

Keeps  patient  profile 

4 

Screens  for  drug  interactions/ 
overlaps 

3 

Serves  as  a learning  tool 

3 

Makes  prescription  refills 
easier 

2 

faster  than  manual 

1 

As  can  be  seen  from  the  table,  the  technicians  particularly 
liked  the  accuracy  of  DataStat,  the  added  safeguards  built 
into  the  system  for  both  the  patient  and  the  user  (who  now 
can  correct  his  mistakes  immediately) , and  the  ease  of  using 
DataStat.  Many  of  the  technicians  were  also  pleased  with  the 
system's  ability  to  type  labels,  keep  profiles,  screen  for 
interactions,  and  at  the  same  time  serve  as  a learning  tool. 

Seven  of  the  technicians  felt  that  their  tasks  in  the 
pharmacy  had  changed  as  a result  of  DataStat.  Most  of  the 
explained  changes  related  to  not  typing  labels,  inputting 
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patient  data  into  the  computer,  and  checking  the  profile  for 
overlaps,  interactions,  or  noncompliance.  Five  respondents 
perceived  their  tasks  to  be  easier  than  before  DataStat,  but 
only  two  felt  that  they  had  less  work  now  with  the  computer. 
Several  technicians,  however,  anticipated  less  work  as  more 
patients  become  logged  into  the  system  and  as  patient  demand 
for  services  recedes. 

Only  one  technician  preferred  the  old  system  to  DataStat, 
giving  the  reason  that  under  computer  operations  there  are 
more  patient  complaints  and  too  much  time  required  for  patient 
education  about  DataStat.  When  asked  what  was  better  with 
the  old  system,  eight  of  the  13  technicians  responded  that 
the  manual  system  was  faster  than  DataStat. 

All  of  the  technicians  agreed  that  DataStat  had  changed 
patient  care  for  the  better,  with  comments  such  as  "a  much 
better  brand  of  service  now,"  "fewer  medications  stocked  up 
at  home  for  children  to  get  into,"  or  "especially  now  with 
profiling  and  screening  for  interactions."  The  consensus 
of  the  technicians,  however,  was  that  most  of  the  patients 
whom  they  had  talked  to  regarded  the  computer  as  a nuisance 
adding  to  their  waiting  time,  until  the  system  was  explained 
to  them  by  a staff  member.  Patients  were  also  confused  by 
the  computer  printing  of  the  generic  drug  name  on  the  label. 

When  asked  if  DataStat  had  changed  the  relationships 
among  the  people  in  the  pharmacy,  three  respondents  reported 
a negative  impact  on  relations  during  the  first  month  when 
technicians  were  not  given  access  to  the  CRTs.  Six  technicians 
perceived  relations  to  be  better  now,  by  having  worked  together 
to  overcome  problems  with  the  system  and  by  having  the  pharma- 
cists out  on  the  front  line  now  with  the  technicians: 
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On  the  old  manual  system,  it  would  be  four  or  five 
technicians  running  the  front  line  and  no  one  else 
would  even  be  around  to  actively  supervise.  It's 
definitely  brought  the  pharmacists  into  more  active 
supervision.  (Technician  #5) 

2 . Equipment  Performance  and  Utility 

The  technicians  were  for  the  most  part  very  pleased  with 
the  performance  of  the  computer  equipment  and  its  ease  of  use. 
All  those  interviewed  reported  having  used  the  CRTs,  and  all 
were  satisfied  with  the  formats  and  amount  of  information  in 
the  displays. 

Terminal  response  times  were  usually  fast  enough  to  suit 
the  technicians,  and  while  two  respondents  perceived  response 
time  to  be  a problem  as  much  as  30  percent  of  the  time,  the 
most  frequent  answer  by  far  was  that  response  time  was  a 
problem  perhaps  5 to  10  percent  of  the  time. 

Every  respondent  observed  an  improvement  in  the  accuracy 
of  label  printing  with  DataStat.  No  other  reductions  in  error 
rates  were  reported  except  for  the  frequent  comment  that  the 
system  won't  accept  mistakes  and  hence  allows  the  user  the 
opportunity  to  correct  his  errors  immediately. 

Without  exception,  the  technicians  interviewed  found 
DataStat  easy  to  use  and  easy  to  learn  to  use,  the  only  dif- 
ficulty being  the  initial  memorizing  of  codes  and  abbrevia- 
tions necessary  for  working  with  the  terminal.  Other  than 
this  initial  obstacle,  the  use  of  terminal  commands,  the 
entry  and  retrieval  of  data,  and  the  changing  of  labels  and 
other  forms  posed  no  problem  whatsoever  for  the  technicians. 
Finally,  the  respondents  were  also  satisfied  with  the  relia- 
bility of  both  the  system  and  its  components,  reporting  only 
scattered  incidents  of  keyboard  failure  or  transmission  prob- 
lems between  NRMC  Charleston  and  the  central  hardware  site. 
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3.  Perceived  Role  Change  and  Suggested  Improvements 


Ten  technicians  interviewed  perceived  a change  in  their 
role  in  the  health  care  delivery  system  as  a result  of  Data- 
Stat,  Typical  descriptions  of  the  nature  of  their  role  change 
were  "less  mechanized  and  closer  to  the  patient,"  "more  aware 
of  potential  problems  and  more  knowledgeable,"  "less  day-to- 
day  drudgery,"  eind  "participating  much  more  in  the  patient 
care  process."  Twelve  of  the  13  respondents  felt  they  were 
more  effective  in  their  roles  as  pharmacy  technicians.  One 
respondent  felt  less  effective  personally  since  "the  computer 
now  does  most  of  what  we  used  to  do,  better." 

Table  6 lists  the  suggestions  for  improvement  of  the 
DataStat  system  offered  by  the  pharmacy  technicians.  The 
most  frequent  suggestions,  for  faster  label  printing  and 
additional  staff,  relate  to  the  technicians'  general  dis- 
satisfaction and  concern  with  patient  waiting  time.  Other 
comments  made  by  more  than  one  technician  were  that  it  would 
be  nice  if  better  user  training  and  instruction  were  provided 
and  if  compounded  prescriptions  and  calling  up  the  profile 
could  be  done  without  having  to  escape  from  the  advanced 
function*  of  DataStat. 

D.  Conclusions 

The  results  of  the  four  pharmacist  interviews  and  13  tech- 
nician interviews  indicate  an  unusually  positive  response  to 
DataStat  after  only  3 months  of  operation.  All  17  respondents 
have  accepted  the  system,  based  on  their  responses  to  the  last 
question  on  the  pharmacist  lEI  and  the  first  question  on  the 
technician  lEI,  and  perceive  it  as  being  beneficial  to  patient 
care  by  improving  the  accuracy  and  capabilities  of  pharmacy 
operations. 

* See  footnote  on  page  8. 


TABLE  6 

TECHNICIAN  SUGGESTED  IMPROVEMENTS  TO 
DATASTAT  OR  ITS  IMPLEMENTATION 


'Almost  all  prescriptions  are  filled  using  what  is  called  the 
advanced  function  of  OataStat.  To  determine  whether 
or  not  a reported  overlap  is  a true  misuse  of  medication, 
the  staff  member  must  escape  from  (leave)  this  function 
and  call  up  the  individual's  profile. 


Other  than  increasing  the  printer  speed  and  the  ability 
of  the  drug  interaction  software  to  discriminate  between 
medications  prepared  for  internal  and  external  use,  the  re- 
spondents found  little  room  for  improvement  in  the  DataStat 
system  itself.  Much  more,  however,  was  offered  in  the  way 
of  suggestions  for  future  installation  of  DataStat  in  mili- 
tary pharmacies.  Foremost  was  the  opinion  of  the  pharmacy 


Response  Cetegory 


Faster  label  printer 
Increase  in  staffing 


Better  user's  manual  and 
training 

More  capabilities  within 
advanced  function* 

Earlier  entering  of  patients 

Earlier  marking  of  ID  cards 
after  entry  into  system 

Less  tiring  color  for  CRT 
display 

Notification  signal  for  when 
labels  are  running  out 

Smaller  terminals  and  printers 


Numbor  of  Rosponios 


staff  that  the  site  should  maintain  the  same  or  greater  number 
of  personnel  in  order  to  hasten  the  system  stabilization  pro- 
cess and  to  fully  utilize  the  new  capabilities  created  by 
the  presence  of  DataStat.  Another  frequent  remark  was  that 
many  headaches  and  patient  complaints  could  have  been  averted 
by  entering  a majority  of  patients  into  the  system  before 
beginning  ADP  operations,  which  in  turn  could  only  be  done 
by  beginning  the  log-in  process  earlier  than  the  2 1/2-month 
interval  at  NRMC  Charleston.  Finally,  two  pharmacists  pointed 
out  that  many  of  the  difficulties  encountered  during  these 
early  months  of  DataStat  operation  stemmed  from  installing 
the  system  at  the  onset  of  the  busy  winter  season  rather 
than  during  the  slower  spring  months. 

As  of  June  30,  1978,  approximately  81,000  patients  (an 
estimated  75  to  85  percent  of  the  local  eligible  population) 
have  been  logged  into  the  system  compared  to  59,000  (50  to 
60  percent)  when  the  interviews  were  conducted  in  March. 

This  fact  and  the  passing  of  the  flu  season  appear  to  have 
reduced  the  problem  with  patient  waiting  time  (based  on  dis- 
cussion with  officers  at  the  pharmacy) , though  no  numerical 
estimates  of  the  decrease  were  obtained.  However,  the  per- 
formance of  DataStat  during  the  month  of  June  was  hampered 
by  a higher  incidence  of  faulty  transmission  between  NRMC 
Charleston  and  the  central  hardware  site  than  when  the  inter- 
views were  conducted.  Problems  with  fully  implementing  the 
DataStat  inventory  functions  and  inpatient  functions  are 
still  being  worked  out,  and  automated  service  has  now  been 
expanded  to  two  nearby  sites — the  Naval  Weapons  Station  Dis- 
pensary (Charleston  Naval  Base),  and  the  U.S.  Air  Force  Base 
Dispensary,  Charleston. 
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II.  INTERVIEW  EVALUATION  INSTRUMENTS 


1 

A.  Introduction 

This  section  contains  the  interview  instruments  used  to  j 

obtain  the  results  presented  in  Section  I.  The  interview  j 

evaluation  instrument  is  designed  to  indicate  the  subjective  | 

perceptions  of  pharmacy  personnel  in  the  salient  areas  of  j 

contrast  between  the  manual  and  automated  modes  of  operation 
in  the  pharmacy.  In  addition,  segments  of  the  lEI  are  de- 
signed to  provide  information  on  the  technical  performance 
of  the  DataStat  system  and  on  its  appropriateness  to  manage-  j 

ment  information  needs  and  system  loads.  The  lEI  is  composed 
of  two  separate  sub-instruments,  which  are  to  be  administered 
independently  to  pharmacists  and  pharmacy  technicians. 


The  principal  technique  used  in  the  lEI  is  controlled 
nondirective  probing.  As  applied  to  information-gathering 
interviews,  this  technique  allows  the  interviewer  to  focus 
on  topics  to  be  discussed  in  a neutral,  or  unbiased,  role. 
Since  this  technique  can  become  inordinately  time  consuming, 
the  nondirective  probes  have  been  supplemented  by  several 
direct  probes.  Application  of  direct  probing  will  be  governed 
by  the  judgment  of  the  interviewer. 
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B.  Instruments 

1 , Pharmacist  lEI 

a.  Drug  Interactions 

(1)  Is  monitoring  drug  interactions  a part  of  your  work  in 
the  pharmacy? 

( ) Yes  

( ) No 

Has  DataStat  changed  the  drug  interactions 
monitoring  system? 

( ) Yes  

( ) No 

Probe:  Is  it  better?  Worse? 

Probe:  Nondirective 

Probe:  What  are  the  benefits 

of  the  DataStat  drug 
interaction  system? 

Probe:  VThat  are  the  disadvan- 

tages of  the  DataStat 
drug  interaction  system? 

(2)  Have  your  tasks  with  respect  to  drug  interactions 
changed  with  the  DataStat  system? 

( ) Yes  \ 

( ) No  ) 


Probe : 

Nondirective 

Probe : 

Which  tasks  have 

changed? 

Probe : 

W^hich  tasks  have 
been  added? 

Probe : 

Which  tasks  have 
been  deleted? 

Probe: 

How  has  your  role 

changed? 

Probe; 

Nondirective 
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(3) 


(4) 


Is  drug  interaction  detection  part  of  your  task? 

( ) 


( ) 


Yes  \ 
No  f 


Probe ; 
Probe : 


Nondirective 


Has  the  DataStat  system 
changed  the  way  you 
perform  these  tasks? 


( ) Yes 


( ) 


No 


Probe ; 
Probe ; 


Nondirective 
Is  your  involvement: 


Same? 

Less? 

Greater? 


Is  drug  allergy  detection  part  of  your  task? 


( ) 


( ) 


Yes 

No 


Probe:  Nondirective 


Probe:  Has  the  DataStat  system  changed  the  way  you 

perform  these  tasks? 


Probe:  In  your  opinion,  will  the  DataStat  system 

change  drug  allergy  detection? 


( ) 


( ) 


Yes 

No 


Probe ! 
Probe ! 


Nondirective 
Is  your  involvement: 


Same? 

Greater? 

Less? 
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BBsassaie: 


(5)  Is  drug  interaction  counseling  part  of  your  task? 

( ) Yes  I 

( ) No  j I 

Probe:  Nondirective 

Probe:  Has  the  DataSLat  system  changed  the  way 

you  perform  these  tasks? 

Probe:  In  your  opinion,  will  the  DataStat  system 

change  drug  interaction  counseling? 

( ) Yes  I 

( ) No  j I 

Probe:  Nondirective 

Probe:  Is  your  involvement: 

Same? 

Greater? 

Less? 

b.  Patient  Profiles 

(1)  Do  you  work  with  patient  profiles? 

( ) Yes 

( ) No  — 1 

(If  respondent  answers  No, 
go  to  section  c . ) 

(2)  Has  the  regimen  of  drug  therapy  for  individual  patients 
changed  with  the  DataStat  system? 

( ) Yes 

( ) No 

Probe:  Nondirective 
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What  changes  have  occurred  in  the 
regimen  of  drug  therapy  due  to  the 
DataStat  system? 


Probe:  Nondirective 


Has  the  amount  of  inforrr.ation  changed? 

How  do  you  think  these  changes  affect 
the  quality  of  health  care? 


Im.proved? 

Worse? 

About  the  same? 


Have  your 
DataStat? 


Probe;  Nondirectiv^e 


Have  these  task  changes  affected  the 
extent  of  your  involvement  in  the 
health  care  delivery  process? 


More  involved? 
Less  involved? 


Has  your  professional  self-image 
changed  as  a result  of  your  patient 
profile-related  tasks? 


( ) Yes 


( ) No  I 

Probe:  Has  your  self- 

image  been: 

Improved? 

Lowered? 


(5)  Has  your  use  of  patient  profiles  changed? 

( ) Yes  ) 

( ) No  * 


Probe:  Nondirective 

(6)  Have  you  discussed  drug  profiles  v/ith  prescribers? 

( ) Yes  

No  — to  section  c. 


( ) 


Do  these  prescribers  ask  you  to  retrieve 
information  from  drug  profiles? 


( ) Yes 


( ) No 

Probe:  Nondirective 

Probe:  !'cw  often? 

Once  a day  or  more? 
Two  or  three  times 
a week? 

Once  a week? 


Do  prescribers  examine  the  drug  profiles  them 
selves? 


( ) Yes  

( ) No 

How  often  do  they  examine 
them? 

Once  a day  or  more? 

Two  or  three  times  a week? 
Once  a week? 


30 


c.  Counseling  and  Compliance 


(1)  Is  monitoring  patient  compliance  with  their  medication 
regimen  part  of  your  work? 


(3) 


( 

) Yes  1 

( 

) No  i 

i 

Probe; 

Nondirective 

Probe ; 

Has  the  OataStat  system  changed  your 
ability  to  detect  overdue  or  early 
refills? 

Probe : 

Nondirective 

Have  your  tasks  with  respect  to  patient  compliance 
monitoring  changed  with  the  DataStat  system? 

( 

) Yes  ^ 

( 

) No  f 

1 

Probe : 

Nondirective 

Probe : 

Which  tasks  have  changed? 

Probe; 

Which  tasks  have  been  added? 

Probe ; 

Which  tasks  have  been  deleted? 

Probe ; 

How  has  your  role  changed? 

Probe ; 

Nondirective  1 

Is 

patient  counseling  part  of  your  work  in  the  pharmacy?  1 

( 

> ’'"“j 

( 

) No  } 

1 

Probe ; 

Nondirective 
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Probe:  Has  the  DataStat  system  changed  your 

counseling  activities? 

Probe:  Nondirective 

Probe:  Is  your  involvement: 

Same? 

Less? 

Greater? 

(4)  Have  you  discussed  patient  compliance  with  prescribers? 
( ) Yes  — 1 


( ) No 


ho  you  perceive  from  your  discussions  that 
prescribers  find  any  change  in  patient 
compliance  since  DataStat? 

I ) Yes  , 


( ) No 


Probe : Better? 

Worse? 


Do  you  perceive  from  your  discussions  that 
prescribers  think  that  the  DataStat  system  has 
the  capacity  to  change  patient  compliance? 

( ) Yes  ) 


( ) NO 


Probe:  Nondirective 


d.  Drug  Information 


(1)  Are  there  prescribers  for  whom  you  regularly  fill 
prescriptions? 

( ) Yes 

{ ) No 

Do  you  talk  with  them  about  the  pharmacy? 

( ) Yes  

( ) No  j—  Go  to  question  (2)  . 

How  often  do  you  talk  with 
them  about  the  pharmacy? 


Probe:  Once  a day  or  more 

often? 

Two  or  three  times 
a week? 

Once  a week? 


Have  you  discussed  drug  interactions  with  them? 
( ) Yes 

( ) No 

Do  you  perceive  from  your 
discussions  that  prescribers 
find  any  change  in  the  fre- 
quency of  drug  interactions? 


(Continued  on  next  page) 


] 
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Do  you  perceive  from  your 
discussions  that  prescribers 
think  the  DataStat  system  has 
the  capacity  to  reduce  the 
frequency  of  drug  interactions? 


( ) Yes 

( ) No 

Will  the  additional  information 
affect  standard  drug  therapy? 

' > I 

{ ) lio  y 

Probe:  Nondirective 

(2)  Has  the  DataStat  system  changed  the  drug  information  system? 

( ) Yes 

( ) No 

Probe:  Nondirective 

Probe;  Capabilities? 

Probe:  More? 

Less? 

Probe:  Nondirective 

Probe:  Accessibility  of  drug  information? 

Probe;  Nondirective 

(3)  Will  the  pharmacy  be  affected  by  the  drug  information  in 
the  DataStat  system? 

( ) Yes 

( ) No 

Probe:  Improved? 

Degraded? 

Probe:  Nondirective 


---  
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Probe:  Benefits  outside  the  pharmacy?  To  society? 
Probe:  Nondirective 
e . Inventory  Controls 

(1)  Has  the  DataStat  system  changed  the  system  for  managing 
stock  levels? 


Yesi 

No  } 

1 

Probe : 

Nondirective 

Probe : 

Better? 

Worse? 

Probe : 

Nondirective 

Probe : 

Has  the  ability  to  accurately  predict 
needed  stock  levels  changed? 

( ) Yes  

( ) No 

Probe:  Improved? 

Worse? 

Probe:  Is  there  a change  in  the  accuracy  of 

seasonal  variation  predictions 
for  stock  levels? 

( ) Yes  

( ) No 

Probe:  Improved? 

Worse? 

Probe:  Has  the  DataStat  system  affected  service? 

( ) Yes  

( ) No 

Probe:  In  what  ways  has 

service  been  affected? 
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Probe;  Nondirective 


Probe:  Better? 

Worse? 

Probe:  Nondirective 

Has  the  eunount  of  outdated  inventory  changed? 

( ) Yes 


( ) No 

Probe:  Nondirective 

Probe;  More?  1 

Probe:  What  is  your  estimate 

of  the  value  of  the 
increased  outdated 
inventory  in  dollars  or 
as  a fraction  of  total 
inventory? 

Probe:  Less?  j 

Probe:  What  is  your  estimate 

of  the  value  of  the 
savings  in  dollars  or 
as  a fraction  of  total 
inventory? 

Has  the  DataStat  system  changed  the  controlled 
substance  inventory  system? 

( ) Yes 


( ) No 

Probe:  Nondirective 

Probe:  Has  the  ability  to  accurately  predict 

needed  controlled  drug  stock  levels 
changed? 

( ) Yes  i 

( ) No  f 

Probe:  Nondirective 


Probe : Improved? 
Worse? 

Probe:  Nondirective 

Probe:  Is  there  a change  in  the  accuracy 
of  seasonal  variation  predictions 
for  controlled  drug  stock  levels? 


( 

) Yes 

j 

( 

) No 

j 

n 

Probe : 

Nondirective 

Probe : 

Improved? 

Worse? 

Probe : 

Nondirective 

f . Outpatient  Dispensing  Methods 

(1)  Has  the  DataStat  system  changed  the  patient  encounter  with 
the  pharmacy  at  the  input  window? 

( ) Yes  J 

( ) No  i I 

Probe:  Nondirective 

Probe:  In  what  ways  has  the  patient  encounter 

changed? 

Probe:  Nondirective 

Probe:  What  are  the  effects  of  each  of  these 
changes? 

Probe : Nondirective 

(2)  Does  the  automatic  label  preparation  change  the  time 
required  foir  label  preparation? 

( ) Yes  i 

( ) No  j I 

Probe:  Nondirective 
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Probe:  Increased? 

Decreased? 

(3)  Is  there  a change  in  accuracy? 

( ) Yes  J 

( ) No  j I 

Probe:  Nondirective 

Probe:  More? 

Less? 

(4)  Is  there  a change  in  the  information  on  the  label? 

( ) Yes  1 

( ) No  f 

Probe:  Nondirective 

Probe:  More? 

Less? 

(5)  Has  the  DataStat  system  changed  the  prescription  check 
process? 

( ) Yes  I 

( ) NO  i 

Probe:  Nondirective 

Probe:  In  what  ways  has  the  prescription 

check  process  changed? 

Probe:  Nondirective 

Probe:  What  are  the  effects  of  each  of 

these  changes? 

Probe:  Nondirective 


(6)  Has  the  DataStat  system  changed  any  other  drug 
dispensing  activities? 


( ) Yes 


( ) No  ) I 

Probe:  Nondirective 

Probe:  What  other  drug  dispensing  activities 
have  changed? 

Probe:  Nondirective 

Probe:  What  are  the  changes,  for  each 
activi ty? 

Probe:  Nondirective 

Probe:  What  are  the  effects  of  each  of  these 

changes? 

Probe:  Nondirective 

(7)  Has  the  DataStat  system  changed  any  other  professional 
activities? 


( ) No 


Yes  \ 

No  ) 

1 

Probe : 

Nondirective 

Probe: 

IVhat  other  professional 
have  changed? 

activities 

Probe : 

Nondirective 

Probe : 

What  are  these  changes, 
activity? 

for  each 

Probe : 

Nondirective 

Probe : 

What  are  the  effects  of 
these  changes? 

each  of 

Probe : 

Nondirective 

g.  General 

(1)  Have  you  talked  with  patients  about  the  DataStat  system? 


( ) Yes 

( ) No 

Probe:  Nondirective 

Probe:  VTliat  did  they  say  about  DataStat? 

(2)  F.ave  you  trained  pharr.acy  personnel  since  DataStat? 

( ) Yes 

( ) No 

Probe:  Is  there  a difference  in  the  training 

time  required  for  new  personnel  to 
work  in  this  pharmacy? 

( ) Yes 

( ) No 

Probe:  Nondirective 

Probe:  More  time  required? 

Less  time  required? 

(3)  Is  there  any  difference  in  error  rates  with  DataStat? 

( ) Yes 

( ) No 

Probe:  With  the  printing  of  reports? 

( ) Yes 

( ) No 

Probe:  Nondirective 

Probe:  More  errors? 

Less  errors? 

Probe:  With  label  printing  compared 
with  manual  preparation? 
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( ) Yes 


( 


Probe:  Nondirective 
Probe:  More  errors? 


Probe:  Less  errors? 

Probe:  Are  there  terminal  character  display 
errors? 


' > I ^ 

( ) No  * Probe:  Nondirective 

(4)  Are  you  familiar  v;ith  the  hardware  capacity  of  the  DataStat 
system  for  consumables  (labels,  computer  paper,  forms)? 

( ) Yes  

( ) No 

Probe:  Is  the  capacity  capable  of  handling  the 

load? 

( ) Yes 

( ) No 

Probe:  Nondirective 

Probe:  Are  changes  necessary? 

Are  changes  desirable? 

Probe:  Nondirective 

(5)  Do  you  use  reports  generated  by  the  DataStat  system? 

( ) Yes 

( ) No 

Probe:  Nondirective 
Probe:  How  often  do  you  use  them? 

Probe:  Do  reports  meet  your  needs? 


Probe : 


Nondirective 


Probe:  Is  format  appropriate? 

Probe:  Nondirective 

Probe:  Any  changes? 

Necessary? 

Desirable? 

Probe:  Nondirective 

Probe:  Is  the  information 

complete? 

( ) Yes  

( ) No 

Probe:  Nondirective: 

Probe:  Too  much  in- 

formation? 

(Go  to  3rd 
probe  below.) 

I 

Probe:  Nondirective 

Probe:  Too  little 

information? 

Probe:  Is  the  time  when  the 

reports  are  generated 
important? 

( ) Yes  I 

( ) No  i 

Probe:  Nondirective 

Probe:  Are  reports 
timely? 


Probe:  Nondirective 

Probe:  Do  you  use  the  reports 

extensively? 

( ) Yes  , 


( ) No 


Probe:  Nondirective 

Probe:  Which  reports? 

1 

Probe:  Nondirective 

Probe:  Which  reports 
do  you  not 
make  use  of? 


(6)  U’ho  uses  the  terminal  displays  in  the  pharmacy? 
Probe:  Pharmacists? 

Probe:  Technicians? 

( > Yes  ) 


( ) No 


Probe:  Have  you  discussed 

the  display  with 
technicians? 

( ) Yes ) 


( ) NO 


Probe:  How  often? 

Probe:  Did  techni- 
cians discuss 
response  time? 

Probe:  Nondirective 

If  response  time  is 
excessive: 


r 


Probe:  Did  they 

complain? 


Probe;  Nondirective 
Probe:  How  often? 

(7)  U’hat  do  you  think  of  the  DataStat  system? 

Probe:  Nondirective 

Probe:  Are  you  satisfied? 

Are  you  dissatisfied? 

Probe:  Nondirective 


t 
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Pharmacy  Technician  lEI 


a . Reaction  to  System 

(1)  Has  the  DataStat  system  changed  your  tasks  in 
the  pharmacy? 

( ) Yes  

( ) No 

Probe:  Nondirective 

Probe:  Are  your  tasks:  Easier? 

Harder? 

Probe:  Do  you  have  more  work? 

Less  work? 

About  the  same? 

Probe:  V'Jhat  are  your  personal  reactions 

to  DataStat? 

Probe:  Nondirective 

■ Probe:  Do  you  like  to  use  the  system? 

( ) Yes 

( ) No 

Probe:  Nondirective 

Probe:  What  do  you  like  about  the  system? 

Probe:  Nondirective 

(Go  to  next  question.) 

Probe:  Nondirective 


I 


I 
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Probe:  Do  you  prefer  the  old  system? 


( ) Yes 

( ) No 


Probe : 

1 

Nondirective 

Probe : 

ViJhat  was  better  with  the  old 
system? 

Probe : 

Nondirective 

Has  the  DataStat  system  changed  patient  care? 

( ) Yes^ 

( ) No 

Probe:  Nondirective 

Probe:  Is  patient  care  better? 

Worse? 

Probe:  Nondirective 

Have  you  talked  with  patients  about  the 
DataStat  system? 

( ) Yes  

( ) No 

Probe:  Did  they  know  the  DataStat  system 

had  been  installed? 


( ) Yes 

( ) No 

Probe:  Did  they  think  it  had  changed 

the  operation  of  the  pharmacy? 


Nondirective 


( ) Yes 

( ) No 

Probe : 

Probe:  Did  they  think  it  was  better? 

Worse? 

Same? 

(If  answer  is  same  go  to  question  (4);  if 
answer  is  better,  worse,  or  mixed,  go  to 
next  probe , ) 

Probe;  Nondirective 

(If  answer  is  better  go  to  next  probe; 
if  answer  is  worse,  skip  next  two  probes; 
if  answer  is  mixed,  skip  next  probe.) 

Probe:  What  is  better? 

(Go  to  question  (4).) 

Probe:  What  is  better? 

Probe:  What  is  worse? 

(4)  Has  the  DataStat  system  changed  the  relation- 
ships among  the  people  in  the  pharmacy? 

( ) Yes 

( ) No 

Probe:  Nondirective 

Probe:  Do  people  get  along  better? 

Worse? 

Probe:  Are  relations  more  relaxed? 

Less  relaxed? 
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b . Equipment/Utility 


(1)  Do  you  use  the  terminal  displays? 

( ) Yes  

( ) No 

Probe:  What  information  do  you  use? 

Probe:  Nondirective 

Probe:  Is  there  information  you  do  not 

use? 

Probe:  Nondirective 

Probe:  Is  there  too  much  information? 

Probe:  Is  the  format  acceptable? 

Probe:  Nondirective 

Probe:  Is  the  format  clear? 

Probe:  Nondirective 

Probe:  Is  the  format: 

Well  organized? 

About  what  you  would  expect? 
Badly  organized? 

Probe:  Nondirective 

Probe:  Are  the  displays  an  appropriate 

means  for  obtaining  the 
information  you  feel  is  required 
for  your  tasks? 

Probe:  Nondirective 

(2)  Are  terminal  response  times  fast  enough? 
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II 


( ) Yes  

( ) No 

Probe;  Always? 

( ) Yes 

( ) No 

Probe:  Nondirective 

Probe:  What  percent  of  the  times  are  not 

fast  enough? 

(3)  Is  there  any  difference  in  the  error  rate  with 
DataStat? 


( 

( 


) Yes  

) No 

Probe:  Printing  of  reports? 

Probe:  Nondirective 

Probe:  More  errors? 

Less  errors? 

Probe:  Label  printing  compared  with 

manual  preparation  of  labels? 

Probe:  Nondirective 

Probe:  More  errors? 

Less  errors? 

Probe;  Terminal  character  display 
errors? 

Probe:  Nondirective 
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(4)  Are  you  familiar  with  the  hardware  capacity  for 
consumables  (computer  paper,  labels,  forms)? 

( ) Yes 

( ) No 

Probe:  Is  the  capacity  sufficient  for 

the  load  placed  on  the  system? 

Probe:  Nondirective 

Probe:  Are  changes  necessary?  Desirable? 

Probe:  Nondirective 

Probe:  What  changes  would  be  desirable? 

(5)  Do  you  replace  consumables? 

( ) Yes 

( ) No 

Probe:  Is  it  easy?  Difficult? 

Probe:  Nondirective 

Probe:  How  easy  (difficult)? 

Probe:  Nondirective 

Probe:  How  much  time  per  day  do  you 

spend  on  this? 

I 


I 


I 
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(6)  Is  the  DataStat  system  easy  to  use? 

( ) Yes 

( ) No 

Probe:  Is  it  difficult? 

Probe:  Nondirective 

Probe:  In  what  ways  is  it  difficult  to 

use  the  system? 

Is  it  easy  to  learn  to  use  the  DataStat  system? 

( ) Yes 

( ) No 

Probe:  Is  it  difficult? 

Probe:  Nondirective 

Probe:  Which  things  were  difficult  to 

learn? 

Are  terminal  commands  easy/difficult  to  use? 

Probe:  Nondirective 

Probe:  ^’Jhat  are  the  difficulties? 

Are  drug  abbreviations  easy/difficult  to  use? 
Probe:  Nondirective 

Probe:  What  are  the  difficulties? 

Is  entry  of  data  easy/difficult? 

Probe:  Nondirective 

Probe:  What  are  the  difficulties? 
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Is  retrieval  of  data  easy/difficult? 

Probe:  Nondirective 

Probe:  What  are  the  difficulties? 

(7)  Is  the  DataStat  system  reliable? 

( ) Yes  1 

( ) No  ( 

Probe:  Does  the  system  go  down? 

( ) Yes  I 

( ) No  j 


Probe:  Nondirective 

Probe:  How  often? 

Probe;  Nondirective 

Probe:  Do  components  break  down? 


( ) Yes  i 

( ) No  7 


Probe:  Nondirective 

Probe:  Which  components  break  down  most 

often? 

Probe:  Nondirective 

Probe:  How  frequently  would  you  say  each 

of  these  components  malfunctions? 
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Perceived  Role 


Has  your  role  in  the  health  care  delivery  system 
changed  with  the  DataStat  system? 

{ ) Yes^ 

— ( ) No 

Probe:  Nondirective 

Probe:  Are  you  more/less  effective? 

Probe;  In  what  ways  has  your  role  changed 
Probe:  Nondirective 
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